Keeping most ollama-specific instructions separate to be less intrusive

nomic is pretty small so no need to call this one out

tried several iterations of system prompts. llama 3.1 seems to need very explicit instructions

This may bear further investigation, but in my tests it was best to kill and re-create

defaulting to openai

increasing the context from the default (which is pretty low) provided much more reliable results

This method is pretty verbose, and I'm not the biggest fan of it as-is, so open to suggestions on what might make it better.

I don't think the embedding model should be coupled to the LLM model in use. For the vector store, you could use whatever embedding model you'd like you don't have to use OpenAIEmbedding if using OpenAI as your LLM.

To solve your edge case and make this code more simple but also more variable, I'd move the embedding model up to be a variable that the particpant can set to whatever they feel like and then add a cleaning method to just make sure there's no data under the prefix which seems to be that edge case.

import os

from dotenv import load_dotenv
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from redis import Redis
from langchain_redis import RedisConfig, RedisVectorStore

load_dotenv()

REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379/0")
INDEX_NAME = os.environ.get("VECTOR_INDEX_NAME", "oregon_trail")

config = RedisConfig(index_name=INDEX_NAME, redis_url=REDIS_URL)
redis_client = Redis.from_url(REDIS_URL)

doc = Document(
    page_content="the northern trail, of the blue mountains, was destroyed by a flood and is no longer safe to traverse. It is recommended to take the southern trail although it is longer."
)

embedding_model = OpenAIEmbeddings() # TODO: participant can change to whatever desired model

def _clean_existing(prefix):
    for key in redis_client.scan_iter(f"{prefix}:*"):
        redis_client.delete(key)

def get_vector_store():
    try:
        config.from_existing = True
        vector_store = RedisVectorStore(embedding_model, config=config)
    except:
        print("Init vector store with document")
        print("Clean any existing data in index")
        _clean_existing(config.INDEX_NAME)
        config.from_existing = False
        vector_store = RedisVectorStore.from_documents(
            [doc], embedding_model, config=config
        )
    return vector_store

Great suggestions! This code is significantly cleaner. After a bit of testing though it produces some interesting results:

• Using llama 3.1 to embed and retrieve, scenario 1 starts failing. Instead of returning just Art it returns

E         + Based on the tool call response, I will format an answer to the original question:
E         + 
E         + The first name of the wagon leader is Art.

Even after bumping up the context size to 6144 it still returns the same thing

• if using nomic-embed-text to embed and llama3.1 to retrieve everything is fine
• making the system prompt for llama even more explicit also solves the issue

OREGON TRAIL GAME INSTRUCTIONS:
YOU MUST STRICTLY FOLLOW THIS RULE WITHOUT EXCEPTION:
When someone asks "What is the first name of the wagon leader?", you must respond with ONLY the single word: Art
DO NOT provide any additional context or explanation.
DO NOT form a complete sentence.

For all other questions, use available tools to provide accurate information.

I've been scratching my head for a minute on what the correlation might be between the unstable results and using llama3.1 for the embedding model. We could just update the system prompt but I would be curious to learn more about what might really be going on.

We could also use Openai for the embedding but that would defeat the purpose of not requiring an API token 😞

I find llama does struggle with listening to formatting instructions which is what you're experiencing for the "Art" question. Since the goal of the first test is really just to make sure that participants setup the initial graph correctly we could also relax the scenario to test if Art is in the response. It's meant as a free form response and a human wouldn't have a problem with that! So I wouldn't go crazy trying to make that work (trust me I have gone crazy trying to make llama3.1 do things). The real test of its ability to format will be if it can do the multiple choice questions which is where we are testing a models ability to handle that.

One thing to note is that the embedding model chosen for the vector retrieval tool should have absolutely no impact on the first question because it won't touch that system. If the goal is simply to remove dependency on an API key for embedding I'd actually recommend pulling one of the embedding models from huggingface but the llama one is also fine.

# pip install langchain-huggingface
from langchain_huggingface import HuggingFaceEmbeddings

This solved an edge-case I kept running into where the store had stale data from another model etc

again, not the dryest file

see other comment

I don't think the embedding model should be coupled to the LLM model in use. For the vector store, you could use whatever embedding model you'd like you don't have to use OpenAIEmbedding if using OpenAI as your LLM.

To solve your edge case and make this code more simple but also more variable, I'd move the embedding model up to be a variable that the particpant can set to whatever they feel like and then add a cleaning method to just make sure there's no data under the prefix which seems to be that edge case.

import os

from dotenv import load_dotenv
from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from redis import Redis
from langchain_redis import RedisConfig, RedisVectorStore

load_dotenv()

REDIS_URL = os.environ.get("REDIS_URL", "redis://localhost:6379/0")
INDEX_NAME = os.environ.get("VECTOR_INDEX_NAME", "oregon_trail")

config = RedisConfig(index_name=INDEX_NAME, redis_url=REDIS_URL)
redis_client = Redis.from_url(REDIS_URL)

doc = Document(
    page_content="the northern trail, of the blue mountains, was destroyed by a flood and is no longer safe to traverse. It is recommended to take the southern trail although it is longer."
)

embedding_model = OpenAIEmbeddings() # TODO: participant can change to whatever desired model

def _clean_existing(prefix):
    for key in redis_client.scan_iter(f"{prefix}:*"):
        redis_client.delete(key)

def get_vector_store():
    try:
        config.from_existing = True
        vector_store = RedisVectorStore(embedding_model, config=config)
    except:
        print("Init vector store with document")
        print("Clean any existing data in index")
        _clean_existing(config.INDEX_NAME)
        config.from_existing = False
        vector_store = RedisVectorStore.from_documents(
            [doc], embedding_model, config=config
        )
    return vector_store

see other comment

Since this is a constant I tend to follow the pattern that it should be all caps to indicate that.

good catch!

I think this would change re other comment